Method and apparatus for determining user input in a three-dimensional field

ABSTRACT

In accordance with one implementation, a method is illustrated that allows a computing device to determine a user input. The method includes detecting one or more user input objects in a 3-dimensional field relative to a 2-dimensional surface. The method also includes determining coordinates for the one or more user input objects relative to the 2-dimensional surface. And, the method further includes determining a user input based on the coordinates.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. §119 of U.S.Provisional Patent Application No. 61/751,958, entitled “ComputerKeyboard That Senses Hovering and Multitouch Gestures Through a Matrixof Proximity Sensors,” and filed on Jan. 14, 2013, which is incorporatedby reference herein in its entirety for all that it discloses or teachesand for all purposes. The present application also claims the benefitunder 35 U.S.C. §119 of U.S. Provisional Patent Application 61/812,824,entitled “Method of Distinguishing Events of Touch and Type Input,” andfiled on Apr. 17, 2013, which is also incorporated by reference hereinin its entirety for all that it discloses or teaches and for allpurposes. The present application also claims the benefit under 35U.S.C. §119 of U.S. Provisional Patent Application 61/814,176, entitled“Interface That Computes Two-Dimensional Coordinates FromThree-Dimensional Input,” and filed on Apr. 19, 2013, which is alsoincorporated by reference herein in its entirety for all that itdiscloses or teaches and for all purposes. The present application alsoclaims the benefit under 35 U.S.C. §119 of U.S. Provisional PatentApplication 61/828,181, entitled “Interface That ComputedTwo-Dimensional Coordinates From Three-Dimensional Input,” and filed onMay 29, 2013 which is also incorporated by reference herein in itsentirety for all that it discloses or teaches and for all purposes. Thepresent application also is a continuation of and claims the benefit ofU.S. Non-Provisional patent application Ser. No. 14/153,793, entitled“User Input Determination,” and filed on Jan. 13, 2014, which is alsoincorporated by reference herein in its entirety for all that itdiscloses or teaches and for all purposes.

BACKGROUND

Over the years, people have developed different ways of communicatinguser input commands to computing devices, such as personal computers.Examples of some devices that have evolved over the years are keyboards,mouse pads, and touch pads, as well as software that converts spokencommands into input commands. Nevertheless, there still remains room forimprovement in the way that users communicate with and efficientlyutilize computing devices.

SUMMARY

In accordance with one implementation, a method is illustrated thatallows a computing device to determine a user input. The method includesdetecting one or more user input objects in a 3-dimensional fieldrelative to a 2-dimensional surface. The method also includesdetermining coordinates for the one or more user input objects relativeto the 2-dimensional surface. And, the method further includesdetermining a user input based on the coordinates.

Another implementation discloses an apparatus that determines a userinput. The apparatus includes a 2-dimensional surface and an objectdetection circuit configured to detect one or more user input objects ina 3-dimensional field relative to the 2-dimensional surface. Inaddition, the object detection circuit is configured to determinecoordinates for the one or more user input objects relative to the2-dimensional surface. Also included is a user input detector configuredto determine a user input based on the coordinates.

In another implementation, one or more computer readable media areprovided. The computer readable media encode computer-executableinstructions for executing on a computer system a computer process. Thecomputer process can include: detecting one or more user input objectsin a 3-dimensional field relative to a 2-dimensional surface;determining coordinates for the one or more user input objects relativeto the 2-dimensional surface, and determining a user input based on thecoordinates.

In one implementation, a device is provided that includes one or morekeys; one or more capacitive sensors disposed in the one or more keys;and an object detector configured to detect one or more user inputobjects in a 3-dimensional field above the one or more keys.

In another implementation, a method is provided that includes receivingsensor data indicative of a touch event on a keyboard; waiting for apredetermined period of time to determine if key press data is received;and signaling a touch event if no key press data is received during thepredetermined period of time while alternatively signaling a key pressevent if key press data is received during the predetermined period oftime. The method can be used for example to discriminate between a touchevent and a type event on a key surface of a keyed device.

In still another implementation, an apparatus is provided that includesa device that includes one or more keys; a plurality of sensors disposedin the one or more keys; and a user input detector configured to receivesensor data indicative of a touch event on the keyed device. The userinput detector is further configured to wait for a predetermined periodof time to determine if key press data is received. The user inputdetector signals a touch event if no key press data is received duringthe predetermined period of time while alternatively signaling a keypress event if key press data is received during the predeterminedperiod of time.

In yet another implementation, one or more computer-readable storagemedia are provided that encode computer-executable instructions forexecuting on a computer system a computer process. The process includesreceiving sensor data indicative of a touch event on a key; waiting fora predetermined period of time to determine if key press data isreceived; and signaling a touch event if no key press data is receivedduring the predetermined period of time while alternatively signaling akey press event if key press data is received during the predeterminedperiod of time.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is neither intended to identify key featuresor essential features of the claimed subject matter nor is it intendedto limit the scope of the claimed subject matter. Other features,details, and utilities, of the claimed subject matter will be apparentfrom the following Detailed Description of various implementations asfurther illustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an apparatus for determining user inputs, inaccordance with one implementation.

FIG. 2 illustrates detection of a user's fingertip and display, inaccordance with one implementation.

FIG. 3 illustrates detection and display of a touch event performed by auser, in accordance with one implementation.

FIG. 4 illustrates detection of a user's fingertips and palm while in“hover” mode, in accordance with one implementation.

FIG. 5 illustrates a flow chart for discriminating between a touch eventand a key event, in accordance with one implementation.

FIG. 6 illustrates a camera serving as a detection sensor above akeyboard, in accordance with one implementation.

FIG. 7 illustrates detection of user input objects without the use of akeyboard as a reference surface, in accordance with one implementation.

FIG. 8 illustrates another example of detection of a user's fingertipsand palm while in hover mode, in accordance with one implementation.

FIG. 9 illustrates a camera sensor that can be used as part of an objectdetection circuit, in accordance with one implementation.

FIGS. 10A and 10B illustrate a key that includes an optical sensordisposed within the key, in accordance with one implementation.

FIG. 11 illustrates tracking of a user input object and display of theuser input object on a display screen, in accordance with oneimplementation.

FIG. 12 illustrates a touch event, in accordance with oneimplementation.

FIG. 13 illustrates a flow chart for a method of determining a userinput, in accordance with one implementation.

FIGS. 14A and 14B illustrate a flow chart for a method of determining auser input, in accordance with another implementation.

FIG. 15 illustrates a block diagram of a system for implementing acomputing device, in accordance with one implementation.

DETAILED DESCRIPTION

Much of the interaction that a computer user has with his or hercomputer is by typing or keying in information and commands via akeyboard. As a result, the user's hands are often positioned in a typingposition. It can be inefficient for the user to have to move a hand to amouse or touch pad or even move a finger to a touch sensitive surface ofa display in order to interface with a graphical user interface on adisplay.

In accordance with one implementation, a user can efficiently interfacewith a graphical user interface by using the space above a surface, suchas the space above a keyboard surface or above another surface, tosignal an input to the graphical user interface. One or more sensors inproximity to the surface detect the position of a user interfaceobject(s) (e.g., a user's finger(s), a stylus, a pen, or some otherpointing device) relative to the surface. The positions of the user'sfingertip(s), for example, are then utilized to determine an input tothe computer. Moreover, the position of the fingertip(s) can betranslated from their position(s) relative to the surface to position(s)on the display. Thus, for example, a fingertip detected above a keyboardsurface can be shown as a circle on a display. As the user moves his orher fingertip above the surface, the movement is detected and the circleis displayed to move in a corresponding manner on the display surface.As the user moves the finger down to the surface, an actual inputsimilar to a mouse click can be generated as an input.

FIG. 1 shows one implementation of a system 100 for implementing a userinput determination system. In accordance with this implementation, auser input object, such as a user's fingertip or stylus, is detected ina 3-dimensional field. Coordinates for the fingertip can then bedetermined and mapped such that the fingertip is displayed as a cursoror other display element on a display. The user's fingertip does nothave to touch a surface—instead, it can simply be detected in free spacewithin the 3-dimensional field. In addition, the user's fingertip cantransition to perform a touch or type event on a keyboard, touch screen,reference surface or other interface.

In FIG. 1, a fingertip 104 of a user is shown disposed above a keyboard106. The tops of the keys of the keyboard define a 2-dimensional surface108 that serves as a reference surface. A 3-dimensional field 107extends above the reference surface and is shown by dashed lines. Theposition of the user's fingertip within the 3-dimensional field isdesignated in accordance with a coordinate system. For example, an x, y,z coordinate system is used in this example. The position of the user'sfingertip in this example is shown to have a position (x₁, y₁, z₁)within the 3-dimensional field.

The 2-dimensional surface corresponds with the 3-dimensional field. Forexample, the 2-dimensional surface can form a reference surface in anx-y plane at the bottom of the 3-dimensional field. In the example ofFIG. 1, the surface delineated by the tops of the keys on the keyboardforms the reference surface. It should be appreciated, however, thatother surfaces could be utilized. For example, the surface of a desk ortable top could be utilized. Even a virtual plane within a threedimensional field could be utilized. Thus, the use of a keyboard in FIG.1 is an example only.

A variety of object detection circuits can be used to detect thepresence of an input object. An object detection circuit can include oneor more sensors, an object detector, a coordinate calculator, aprocessor, and a memory, for example. FIG. 1 shows a camera 110 that canbe used as part of an object detection circuit to detect a user'sfingertip. Other sensors, such as capacitive sensors in the keys of akeyboard may be used, as well. The sensors generate raw data that isused by an object detector 121 to determine the presence of user inputobject(s). When one or more sensors generate data indicating that one ormore objects is in a sensor field, the object detector can signal that auser input object(s) has entered the sensor field. The sensor data canbe converted by a coordinate calculator 122 to provide coordinateswithin the 3-dimensional field 107 (shown by dashed lines and the topsurface of the keyboard) for any detected object(s). A processor 130 canbe used by the coordinate calculator to convert the sensor data intocoordinate data. In some implementations, the sensor(s), objectdetector, coordinate calculator, and processor can be configured as anintegrated unit.

A computing device 120 is shown in FIG. 1 for converting the datagathered by the sensor(s) into display data for a display. For example,FIG. 1 shows a mapper 124. The mapper is configured to map coordinatescorresponding to the position of at least one of the input objectsrelative to the 2-dimensional surface to the display surface. Forexample, the mapper can take X, Y, Z coordinate data that was obtainedby the coordinate calculator and convert that data into display data.FIG. 1 shows a display element 142 displayed in a particular position onthe display screen. The position of the display element is based uponthe originally determined x₁, y₁ coordinates of the user's detectedfingertip while the size of the circle that is displayed is based uponthe previously determined z₁ coordinate.

For example, display elements can be rendered by scaling the width andheight of a rectangular plane of the 3-dimensional field to have thesame width-to-height ratio as a display. In this manner, the coordinatesof user input objects can be translated to display coordinates for adisplay element.

A user input detector 126 determines what to do with the data producedby the mapper. In the example shown in FIG. 1, the data is communicatedto a display controller 128 that displays circle 142 on the display.

The user input detector can also be used to detect, for example, usercommands, gestures, touch events, and type events, as well asmulti-finger or single-finger modes. For example, the user inputdetector can analyze a sequence of fingertip coordinate data anddetermine a gesture that has been performed by the user. If the gesturematches a predefined gesture, the user input detector can signal acommand that corresponds with that predefined gesture. As anotherexample, the user input detector can monitor the coordinate dataassociated with a fingertip. If the fingertip is placed at the referencesurface position for a predefined period of time, the user inputdetector can signal that a touch event has occurred. Moreover, the userinput detector can also signal a change in modes of operation (e.g.,from fingertip input to keyboard input) when a fingertip is placedagainst the reference surface for an extended period of time.

The object detector in the example of FIG. 1 can limit the sensor fieldto a size in the x-y plane that is proportional to the size of thedisplay. This allows the user to experience an absolute-position system,as opposed to a relative-position system. An absolute-position systemallows the user to place a user input object(s) into the sensor fieldand to see the user input object(s) displayed on the display in thecorresponding position of the display. A relative-position system suchas a mouse or touchpad would require the user to move a cursor to thedesired location on a display. Thus, an absolute-position system is veryintuitive for a user to implement.

Moreover, the system implemented by the apparatus in FIG. 1 allows aplane of touch to be separate from a plane of display (or view). Whileabsolute-position sensing is utilized, absolute-positioning is performedin a plane separate from the plane of the display.

FIG. 2 shows an example of a user interacting with a computer system 200that implements a user input determination system. In FIG. 2, a user'sfinger is positioned above a keyboard surface. The keyboard surfaceforms a reference surface. As the user moves his or her finger throughthe space above the keyboard, the user's fingertip is detected by objectdetector(s) located in keys of the keyboard. Corresponding display datais generated and displayed on the display screen. As can be seen in FIG.2, the X-Y position of the user's fingertip relative to the keyboardcorresponds with the position of the circle displayed on the displayscreen.

FIG. 3 shows detection of a touch event performed by the user. In FIG.3, the user's fingertip has come into contact with the keyboard of acomputer system 300 for a predetermined period of time. This can simplybe caused by the user's fingertip resting on the keyboard surface for apredetermined period of time. In this example, the user inputdetermination system displays a dot on the display screen. Because theproximity of the user's fingertip has been detected to be at thereference surface, the user input determination system can determinethat a touch event has occurred.

Such a touch event can be used by a user to interact with a graphicaluser interface. For example, if a graphical user interface is displayinga list of songs for selection, the user can move his or her fingertiprelative to a reference surface, e.g., a keyboard, while watching thecorresponding display element on the display. As the display elementmoves over the desired song, the user can touch the reference surface,e.g., the keyboard, to perform a touch event that selects the song.

FIG. 4 shows a computer system 400. FIG. 4 illustrates that multipleuser input objects can be detected at the same time. In FIG. 4, all thefingertips and the palm of a user's hand are positioned above a keyboardthat is serving as a reference surface. The fingertips and palm are nottouching the keyboard but are instead in the 3-dimensional field abovethe keyboard. This 3-dimensional field is delineated by the dashed linesand the surface of the keyboard. The user input determination systemdetects and displays the user input object information on the display assix circles. Because the palm of the user's hand is closer to thereference surface, the palm is displayed as a circle having a smallerdiameter than the circles corresponding to the user's fingertips.

FIG. 4 also illustrates a “hover” mode for the user input determinationsystem. Hover mode is when the user input determination system isdetecting objects in the 3-dimensional field above a reference surface.During hover mode, detected user input objects can be displayed on thedisplay. In addition, the user input determination system can alsooperate in “touch” mode and “type” mode. Touch mode is when the userinput determination system detects and responds to a touch event. Typemode is when a depression of a key on a user interface device isdetected and registered as an intended key depression event.

A touch event is triggered when a user's fingertip(s) (or other inputobject(s)) are placed in an interactive area. The interactive area couldbe the surface of a computer keyboard or even just a predetermined levelof a 3-dimensional field.

A key-press event is triggered when a key is pressed down. If the keypress event is triggered within a short period of time after thetriggering of a touch event, type mode can be initiated. Alternatively,touch mode is maintained.

While in touch mode, pressing down of keys by a user need not trigger akey-press event. Thus, accidental pressing down of keys during touchmode for a short period of time will not terminate touch mode andinitiate type mode. Rather a key-press event has to be for a sufficientpre-determined duration and with sufficient depression distance to makeit clear that a key-press event is intended.

In accordance with one implementation, a method of discriminatingbetween a touch event and a type event can be utilized. The method isinitiated by operation 502 and includes reading an input, as shown byoperation 504. A decision operation 506 determines whether a touch inputhas been detected. If the input that has been read does not match atouch input, then a new input reading is made. If the input reading isdetermined to be a touch input, however, then a timer can be initiatedin operation 508. The timer can run for a few milliseconds. Decisionoperation 510 queries a routine to determine whether a key press eventis detected. Decision operation repeats this routine until an intervalcount is reached. In operation 516, keyboard input(s) are read. Indecision operation 518, if no key press is detected by the keyboardinput, then the process can be repeated. If a key press is detected,then the interval timer can be stopped as shown by operation 520 and akeyboard event can be signaled as shown by operation 522. Thus, thekeyboard input would override a touch event. If the decision operationin decision block 510 expires due to elapsed time or elapsed number ofallocated iterations, the timer is stopped as shown by operation 512 anda touch event is signaled as shown by operation 514. After a touch eventor type event is signaled, the process can begin again from operation504. This implementation allows the same keypad, keyboard, or otherdevice with key(s) to serve as both a touch and a type interface. Thisimplementation can be accomplished because the system allows a typeinput to supersede a touch input when a type input is detected.

The flow chart shown in FIG. 5 can also be used to select modes ofoperation for a device. When a touch event is detected, a device can beplaced into touch mode. When a key press event (sometimes referred to asa type event) is received, a device can be placed into type mode. Thesemodes can be triggered by the user input detector described above. Whena key press event is determined, the user input detector can place adevice in type mode. When a touch event is determined, the user inputdetector can place the device in touch mode.

The sensor data discussed in FIG. 5 can be provided by a variety ofdifferent sensors. For example, the sensor data could be gathered by acamera sensor(s), by a capacitive sensor(s), by an optical sensor(s), orby other types of sensor.

It should also be noted that a change in mode can be implemented by akeyboard shortcut or by a gesture defined by either the user orpre-defined by the system.

Another change in mode that can be implemented is a change fromsingle-finger mode to multi-finger mode. Performing a touch operationwith one finger allows a system to determine single-touch gestures,e.g., click, swipe, and drag, that are to be utilized. Performing atouch operation with multiple fingers allows a user to performmulti-touch gestures, such as pinch, multiple-fingers-swipe, and pivotalrotation. Thus, in one implementation, the system provides both amulti-finger mode and a single-finger mode. Multi-finger mode isinitiated when coordinates of more than one fingertip (or other userinput object) are detected. Single-finger mode is initiated whencoordinates of just a single fingertip (or other user input object) aredetected. While in multi-finger mode, the hover feature can be disabled.The hover feature can be maintained when the system operates insingle-finger mode. Moreover, while in multi-finger mode, a single touchevent can be dispatched when a single finger is placed in contact with atouch surface, while a multi-finger touch event is dispatched whenmultiple fingers are placed in contact with a touch surface.

FIG. 6 shows an example of a system 600 where a camera system is used asa sensor. The camera sensor is shown coupled with the top of a displayscreen and oriented at a keyboard surface. The data gathered by thecamera can be interpreted to generate position data for user inputobjects, e.g., a user's fingertips and palm, that are in the3-dimensional field above a reference surface, such as a keyboardsurface. FIG. 6 shows that the fingertip and palm data as it relates tothe 2-dimensional reference surface is displayed as display elements onthe display screen.

In order to track the fingertip positions in 3-dimensions, a variety oftechniques can be used with a camera sensor. For example, stereoscopicdisparity mapping, time-of-flight depth mapping, and structured depthmapping may be used.

For example, with stereoscopic cameras, a determination of the 3Dstructure of a scene or 3D coordinates of objects in the scene can bemade using two or more images of the 3D scene, each acquired from adifferent viewpoint in space. The images are simultaneously analyzed tocalculate disparity (distance between corresponding points when the twoimages are superimposed) either for every point in an image (a disparitymap) or for specific points (e.g., fingertips). In addition to the x andy coordinates, which are readily available from the images, z (or depth)can be calculated by using disparity as a measure of distance away fromthe cameras (the further an object, the smaller the disparity).

It is not necessary that a keyboard be used as a reference surface. Inthe computer system 700 of FIG. 7, a display is shown disposed on asupporting structure, such as a desk surface. A 3-dimensional field isshown extending above the desk surface. In this example, the desksurface serves as a 2-dimensional reference surface. Thus, as the user'shand hovers above the desk surface in front of the display, the user'sfingertips and palm are displayed as display elements on the display.

FIG. 8 illustrates that a camera sensor can be placed in differentpositions to detect user input. FIG. 8 shows a system 800 in which acamera 802 disposed beneath a display screen 804. The camera is orientedat the 3-dimensional field in front of the display screen. Again, inthis example, the 2-dimensional reference surface can be any surface.The 2-dimensional surface can even be a virtual plane in space, such asan X-Y plane in an X, Y, and Z coordinate system. Thus, the referencesurface need not be an actual physical surface.

FIG. 9 illustrates an example of a clip-on camera 900 that can be usedas an object detection sensor. The camera shown in FIG. 9 can be clippedon to a computing device, such as a display screen of a laptop computer.Alternatively, the camera can be placed to the side of an area where auser chooses to perform user inputs.

Other types of sensors besides a camera(s) can be used as well to detectuser input objects. FIG. 10A illustrates an example of an optical sensordisposed in a key of a keyboard. The optical sensor is shown having fouroptical proximity sensors soldered to a multiplexer board. Rectangularopenings are cut in the keycap in order for the sensors to sense abovethe surface of the key. FIG. 10B shows such circuitry with the cap ofthe key removed. The optical sensor detects when objects are disposedabove it. A grid of these optical sensors can be disposed in a grid ofkeys on a keyboard. Thus, as a user moves his or her fingertip above thegrid, the sensor that senses the strongest signal will indicate that theuser's fingertip is closest to the position of that particular sensor.The system can then compute the X and Y coordinates for the position ofthe user's fingertip because the system knows the X and Y coordinates ofthe key in which that particular optical sensor is located. For example,the system can triangulate a more accurate position using signals frommultiple sensors. A Z value for the position of the user's fingertip canbe determined by comparing the measured optical sensor output value withpredetermined calibration data for that optical sensor. The measured Zvalue data is sometimes referred to as proximity lines. Again, aproximity value can be determined by triangulation using sensor datafrom multiple keys. For example, these techniques can be utilized by acoordinate calculator.

Capacitive sensors can also be used. In one implementation, capacitivesensors can be used within keys of a keyboard. The capacitive sensorscan be placed in all keys or a selected group of the keys. As user inputobjects move above the capacitive sensors, sensor data can be collected.This sensor data may then be analyzed, for example, by a coordinatecalculator that determines the coordinates of a user input object(s)relative to the keys.

In one implementation, capacitive sensors can be disposed only in keysthat form a layout that is proportional to a display surface. In anotherimplementation, the capacitive sensors can be disposed in a layout thatis not proportional to the display surface. The sensors can be used togather data that is used to compute coordinates of user input itemspresent in proximity to the keyboard.

In another implementation, a capacitive sensor grid layout can bedisposed underneath a touch pad. The touch pad can be separate from orintegrated with a computer or keyboard. Sensors underneath the touch padcan be used to sense proximity of a user input object relative to thesensor grid. The data may then be used to calculate coordinate data forthe user input object(s). Capacitive sensors are available, for example,from Cypress Semiconductor or San Jose, Calif.

A processor, memory, sensor, object detector, and coordinate calculatormay be integrated with a keyboard. In such an implementation, thekeyboard itself can generate coordinate data for a user input object(s).

Other sensors may be used, as well. For example, ultrasonic sensors canbe used as sensors. Moreover, micro-electro-mechanical system (MEMS)devices can be used to fabricate sensors that can be disposed in keys orother elements. Still other types of sensors can be used, as well.

When a keyboard is utilized as a sensor grid, not all of the keys needto contain sensors. However, the resolution of a sensor grid matriximplemented via sensors disposed on a keyboard can be increased byincreasing the number of sensors of the sensor grid matrix. Since keyson a standard QWERTY keyboard are not arranged in strict columns so asto correspond with the shape of a display device, the system can beconfigured to interpolate the interactive area delineated by the sensorsto the dimensions of the display device. As one example, the interactivearea can be defined to be the largest rectangular region that fitswithin the bounds of the block of sensor keys—assuming that the displayscreen is also rectangular.

The dataset captured by sensor(s) can be presented in a graphical formatthat resembles a traditional heat map. This heat map allows detection ofmultiple inputs simultaneously. For example, a heat map shows an imageof a top-down view of proximity sensor locations within an interactivearea defined by a sensor grid. Locations closest to the user inputobjects, e.g., the user's fingertips, show the reddest hues, forexample. Other locations show hues that fade into bluer colors where thesensor data is less pronounced. The positions of multiple fingers can becomputed either directly or from raw data using simple statisticaltechniques. The positions can also be computed from the heat map usingcomputer vision techniques, such as blob detection. These techniques canbe implemented by the coordinate calculator of the system.

In accordance with one implementation, a system is calibrated by aparticular user. In this example implementation, the user places his orher fingertips or other objects on a specified number of points in a3-dimensional field. For example, the user can touch four points on areference surface. According to the user's preference, parametersincluding but not limited to sizes, shapes, colors, and transparencylevels of display elements, such as touch cursors, can be selected.During use of the system these parameters are used to indicateproximities and/or positions of fingertips or objects relative to areference surface.

FIG. 11 illustrates a computer system 1100 and how a user can cause adisplay element to move across a screen. In FIG. 11 a user's fingertipis detected above the lower left-hand corner of a reference surface. Thereference surface is shown in dashed lines. The user's hand andfingertip then move across the reference surface. As the user's hand andfingertip move across the reference surface, an intermediate position issensed. Eventually, the user's hand and fingertip come to rest at aposition above the upper right-hand corner region of the referencesurface. FIG. 11 shows that display elements are shown on the displaythat correspond to the movement of the user's fingertip. Displayelements corresponding to previous positions of the user's fingertip canbe made to slowly fade from the display while the current positionassociated with the user's fingertip is displayed in bold outline on thedisplay.

FIG. 12 illustrates another example of how a display element can bechanged in response to a change in proximity of a user input object,such as a user fingertip, to a reference surface. The screen 1220 showsa display element 1204 as a circle when a fingertip is positioned abovea reference surface. The display element is changed to a circle ofsmaller diameter as the fingertip is brought closer to the referencesurface. When the fingertip is placed very close to the referencesurface, the display element becomes a dot 1206.

While a diameter of a circle is used as the varying parameter of thedisplay element in FIG. 12, it should be noted that other parameterscould be utilized. For example, a change in size, shape, color,brightness, and transparency could be utilized, among others. Someexample transformations include: alpha (transparency index) levels of adisplay element are inversely proportional to a user input object'scorresponding proximity; widths and heights of display elements aredirectly proportional to a user input object's corresponding proximity;and brightness levels of display elements are directly proportional to auser input object's corresponding proximity. As another example, when auser's fingertips are hovering above a reference surface, thecorresponding display elements are semi-transparent circles of a certaincolor. When the user's fingertips are in contact with a touch surface,the display elements shrink to dark, fine points.

In FIG. 13, a process of determining a user input is illustrated by aflow chart. In operation 1302, one or more user input objects in a3-dimensional field are detected relative to a 2-dimensional surface. Inoperation 1304, coordinates corresponding to a position of at least oneof the user input objects relative to the 2-dimensional surface aredetermined. And, in operation 1306, a user input is determined based onthe coordinates.

Another implementation of determining a user input is illustrated byFIGS. 14A and 14B. In operation 1402, one or more user input objects aredetected in a 3-dimensional field relative to a 2-dimensional surface.In operation 1404, coordinates for one or more user input objects aredetermined. For example, x, y, and z coordinates can be determined inaccordance with an x, y, and z coordinate system. Moreover, the x, y,and z coordinates can be determined relative to the 2-dimensionalsurface. Other coordinate systems can also be utilized, such as aspherical coordinate system

In operation 1406, the coordinates corresponding to the position of atleast one of the user input objects is mapped to a display surface. Forexample, x, y, and z coordinates of a user input object can be mapped toa position for display on a display screen. Moreover, the z coordinatecan be used to select the size of a display element to use at thecalculated position on the display.

Once coordinates are determined for a user input object, various modesof operation can be determined. For example, operation 1408 shows thatthe coordinates can be used to determine whether a hover event is takingplace. A hover event would be determined if the user interface object islocated above a reference surface but not touching the referencesurface.

When one or more user interface objects are detected, the system canfocus on a particular user input object. For example, when multiplefingertips are detected, the system can disregard some of the user inputobjects and focus on just one. This is illustrated by operation 1410.

Operation 1412 shows that a touch event can also be detected. A touchevent can be detected when a user input object is detected to be at atouch surface for a predetermined period of time. The touch surface cancoincide with a physical surface, such as a touchpad or keypad.Alternatively, if no physical input device is available, the touch eventcan be determined by the user input being detected at an inert surface,such as a table top. The touch event can even be determined by the userinput object being present at a predetermined position in space for apredetermined amount of time.

When a touch event is detected, the system can turn off hover mode andinput a command indicated by the touch event. This feature isillustrated by operation 1414.

The coordinates of a user input object can also be used to signal a userinput. For example, if the system is in hover mode, a display elementcorresponding to a user input object can be displayed on the displaysurface. This is illustrated by operation 1418.

In operation 1420, a characteristic of a display element can be variedbased on the proximity of a user input object to the 2-dimensionalsurface. For example, operation 1422 shows that the diameter of a circleused as the display element can be varied depending on how proximate theuser input object is to the 2-dimensional surface.

One aspect of the user input determination system is that user inputscan be communicated without a user having to touch a device. Operation1424 illustrates that a command can be determined from a user withoutthe user touching a keyboard, a mouse, a touchpad, a display, or otherphysical device in order to issue a command. For example, 3-dimensionalgestures can be determined from the coordinates that a user input objectmoves through during a predetermined period of time. This is illustratedby operation 1426.

FIG. 15 discloses a block diagram of a computer system 1500 suitable forimplementing aspects of at least one implementation. The computer system1500 can be used to implement one or more components of the user inputdetermination system disclosed herein. For example, in oneimplementation, the computer system 1500 can be used to implement thecomputing system 120, display 140, camera 110, and keyboard 108 of FIG.1.

As shown in FIG. 15, system 1500 includes a bus 1502 which interconnectsmajor subsystems such as a processor 1504, internal memory 1506 (such asa RAM or ROM), an input/output (I/O) controller 1508, removable memory(such as a memory card) 1522, an external device such as a displayscreen 1510 via a display adapter 1512, a roller-type input device 1514,a joystick 1516, a numeric keyboard 1518, an alphanumeric keyboard 1520,smart card acceptance device 1524, a wireless interface 1526, and apower supply 1528. Many other devices can be connected. Wirelessinterface 1526 together with a wired network interface (not shown), canbe used to interface to a local or wide area network (such as theInternet) using any network interface system known to those skilled inthe art.

Many other devices or subsystems (not shown) can be connected in asimilar manner. Also, it is not necessary for all of the devices shownin FIG. 15 to be present to practice an implementation. Furthermore, thedevices and subsystems can be interconnected in different ways from thatshown in FIG. 15. Code to implement one implementation can be operablydisposed in the internal memory 1506 or stored on storage media such asthe removable memory 1527, a floppy disk, a thumb drive, a CompactFlash®storage device, a DVD-R (“Digital Versatile Disc” or “Digital VideoDisc” recordable), a DVD-ROM (“Digital Versatile Disc” or “Digital VideoDisc” read-only memory), a CD-R (Compact Disc-Recordable), or a CD-ROM(Compact Disc read-only memory). For example, in an implementation ofthe computer system 1500, code for implementing the user inputdetermination system can be stored in the internal memory 1506 andconfigured to be operated by the processor 1504.

In the above description, for the purpose of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the implementations described. It will be apparent,however, to one skilled in the art that these implementations can bepracticed without some of these specific details. For example, whilevarious features are ascribed to particular implementations, it shouldbe appreciated that the features described with respect to oneimplementation can be incorporated with other implementations as well.By the same token, however, no single feature or features of anydescribed implementation should be considered essential, as otherimplementations can omit such features.

In the interest of clarity, not all of the routine functions of theimplementations described herein are shown and described. It will, ofcourse, be appreciated that in the development of any such actualimplementation, numerous implementation-specific decisions must be madein order to achieve the developer's specific goals, such as compliancewith application—and business-related constraints, and that thosespecific goals will vary from one implementation to another and from onedeveloper to another.

According to one implementation, the components, process steps, and/ordata structures disclosed herein can be implemented using various typesof operating systems (OS), computing platforms, firmware, computerprograms, computer languages, and/or general-purpose machines. Themethod can be run as a programmed process running on processingcircuitry. The processing circuitry can take the form of numerouscombinations of processors and operating systems, connections andnetworks, data stores, or a stand-alone device. The process can beimplemented as instructions executed by such hardware, hardware alone,or a combination thereof. The software can be stored on a programstorage device readable by a machine.

According to one implementation, the components, processes and/or datastructures can be implemented using machine language, assembler, C orC++, Java and/or other high level language programs running on a dataprocessing computer such as a personal computer, workstation computer,mainframe computer, or high performance server running an OS such asSolaris® available from Sun Microsystems, Inc. of Santa Clara, Calif.,Windows 8, Windows 7, Windows Vista™, Windows NT®, Windows XP PRO, andWindows® 2000, available from Microsoft Corporation of Redmond, Wash.,Apple OS X-based systems, available from Apple Inc. of Cupertino,Calif., BlackBerry OS, available from Blackberry Inc. of Waterloo,Ontario, Android, available from Google Inc. of Mountain View, Calif. orvarious versions of the Unix operating system such as Linux availablefrom a number of vendors. The method can also be implemented on amultiple-processor system, or in a computing environment includingvarious peripherals such as input devices, output devices, displays,pointing devices, memories, storage devices, media interfaces fortransferring data to and from the processor(s), and the like. Inaddition, such a computer system or computing environment can benetworked locally, or over the Internet or other networks. Differentimplementations can be used and can include other types of operatingsystems, computing platforms, computer programs, firmware, computerlanguages and/or general purpose machines. In addition, those ofordinary skill in the art will recognize that devices of a less generalpurpose nature, such as hardwired devices, field programmable gatearrays (FPGAs), application specific integrated circuits (ASICs), or thelike, can also be used without departing from the scope and spirit ofthe inventive concepts disclosed herein.

The above specification, examples, and data provide a completedescription of the structure and use of exemplary implementations. Sincemany implementations can be made without departing from the spirit andscope of the invention, the invention resides in the claims hereinafterappended. Furthermore, structural features of the differentimplementations can be combined in yet another implementation withoutdeparting from the recited claims.

What is claimed is:
 1. A method of determining user input in a 3D fieldrelative to a 2D surface, the method comprising: detecting one or moreuser input objects in a 3-dimensional field relative to a 2-dimensionalsurface; determining coordinates for the one or more user input objectsrelative to the 2-dimensional surface; determining a user input based onthe coordinates.
 2. The method of claim 1 and further comprising:mapping coordinates corresponding to the position of at least one of theinput objects relative to the 2-dimensional surface to a displaysurface.
 3. The method of claim 1 wherein the detecting one or more userinput objects comprises: utilizing capacitive sensors in keys of akeyboard to detect the one or more user input objects.
 4. The method ofclaim 3 wherein the keys of the keyboard delineate the 2 dimensionalsurface.
 5. The method of claim 1 and further comprising: utilizing atleast one camera to detect the one or more user input objects.
 6. Themethod of claim 1 and further comprising: utilizing optical sensors inkeys of a keyboard to detect the one or more user input objects.
 7. Themethod of claim 6 wherein the keys of the keyboard delineate the 2dimensional surface.
 8. The method of claim 1 wherein determining a userinput comprises determining a hover event.
 9. The method of claim 1wherein determining a user input comprises determining a touch event.10. The method of claim 1 and further comprising: turning off a hovermode when a touch event is detected.
 11. The method of claim 1 andfurther comprising: displaying a user input on the display surface. 12.One or more computer-readable storage media encoding computer-executableinstructions for executing on a computer system a computer process, thecomputer process comprising: detecting one or more user input objects ina 3-dimensional field relative to a 2-dimensional surface; determiningcoordinates for the one or more user input objects relative to the2-dimensional surface; determining a user input based on thecoordinates.
 13. The one or more computer-readable storage media ofclaim 12 wherein the computer process further comprises: mappingcoordinates corresponding to the position of at least one of the inputobjects relative to the 2-dimensional surface to a display surface. 14.The one or more computer-readable storage media of claim 17 wherein thedetecting one or more user input objects comprises: utilizing capacitivesensors in keys of a keyboard to detect the one or more user inputobjects.
 15. The one or more computer-readable storage media of claim 12wherein the computer process further comprises: utilizing at least onecamera to detect the one or more user input objects.
 16. The one or morecomputer-readable storage media of claim 12 wherein the computer processfurther comprises: utilizing optical sensors in keys of a keyboard todetect the one or more user input objects.
 17. The one or morecomputer-readable storage media of claim 12 wherein determining a userinput comprises determining a hover event.
 18. The one or morecomputer-readable storage media of claim 12 wherein determining a userinput comprises determining a touch event.
 19. The one or morecomputer-readable storage media of claim 12 wherein the computer processfurther comprises: turning off a hover mode when a touch event isdetected.
 20. The one or more computer-readable storage media of claim12 wherein the computer process further comprises: displaying a userinput on the display surface.
 21. An apparatus comprising: a2-dimensional surface; an object detection circuit configured to detectone or more user input objects in a 3-dimensional field relative to the2-dimensional surface and configured to determine coordinates of one ormore input objects relative to the 2-dimensional surface; and a userinput detector configured to determine a user input based on thecoordinates.
 22. The apparatus as claimed in claim 21 and furthercomprising: a display surface; and a mapper configured to mapcoordinates corresponding to the position of at least one of the inputobjects relative to the 2-dimensional surface to the display surface.23. The apparatus as claimed in claim 21 wherein the 2-dimensionalsurface comprises a keyboard comprising keys having capacitive sensors.24. The apparatus as claimed in claim 21 wherein the object detectioncircuit comprises one or more cameras.
 25. The apparatus as claimed inclaim 21 wherein the object detection circuit comprises a keyboardcomprising keys having optical sensors.
 26. The apparatus as claimed inclaim 21 wherein the user input detector is configured to determine ahover event.
 27. The apparatus as claimed in claim 21 wherein the userinput detector is configured to determine a touch event.
 28. Theapparatus as claimed in claim 31 and further comprising a displaycontroller configured to display user input on the display based on thecoordinates.
 29. The apparatus as claimed in claim 28 wherein thedisplay controller is further configured to vary a display elementcharacteristic based on a change in a proximity of at least one inputobject to the 2-dimensional surface.
 30. The apparatus as claimed inclaim 29 wherein the display controller is configured to vary a diameterof a circle used for the display element.